Computer-implemented method for the use of stored specification parts

ABSTRACT

A computer-implemented method for the use of stored specification parts of at least one test and/or simulation, comprising the steps: providing the at least one test to be specified and/or the one simulation for testing driving functions of a vehicle and the at least one test and/or the at least one simulation are determined by at least one parameter value and/or setting value; and performing a specification of the at least one test and/or the at least one simulation, wherein the specification comprises at least one specification part, wherein the parameter values and/or the setting values are selected and concrete parameter values and/or setting values are assigned by the specification for the test and/or the simulation, wherein the parameter values and/or setting values are selected manually or automatically, and wherein already stored specification parts are selected manually and/or automatically and integrated into the specification.

This nonprovisional application claims priority under 35 U.S.C. § 119(a)to German Patent Application No. 10 2022 114 913.7, which was filed inGermany on Jun. 14, 2022, and which is herein incorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates to a computer-implemented method for the use ofstored specification parts. The present invention further relates to atest unit for the use of stored specification parts. The presentinvention moreover relates to a computer program and to acomputer-readable data storage medium.

Description of the Background Art

Driving assistance systems such as, e.g., adaptive cruise control and/orfunctions for highly automated or autonomous driving can be verified orvalidated using various verification methods. In this regard, inparticular, hardware-in-the-loop methods, software-in-the-loop methods,simulations, and/or test drives can be used.

The effort, in particular the time and/or cost, for testing such vehiclefunctions using the above-mentioned verification methods is typicallyvery high, because a large number of potentially possible drivingsituations must be tested. A specific test case and/or test suites mustbe specified if applicable for each of these driving situations. Testsuites here represent a combination of a number of related test cases.

Testing an at least partially autonomous transport vehicle exclusivelyon the road with travel distances of over billions of kilometers is notpossible for reasons of time and cost. In addition, many redundant testkilometers would arise, whereas critical and unusual situations that arerelevant to the capabilities of the at least partially autonomousvehicle, however, would not occur. For this purpose, precisespecifications must be generated for these scenarios and the test casesthey contain.

This can lead to a high effort for test drives as well as forsimulations. DE 10 2017 200 180 A1 provides a method for verifyingand/or validating a vehicle function intended for autonomously guiding avehicle in the longitudinal and/or lateral direction.

The method in DE 10 2017 200 180 includes a step for determining, on thebasis of environmental data relating to an environment of the vehicle, atest control instruction of the vehicle function to an actuator of thevehicle, wherein the test control instruction is not implemented by theactuator. The method further includes simulating, on the basis ofenvironment data and using a road user model regarding at least one roaduser in the environment of the vehicle, a fictitious traffic situationthat would exist if the test control instruction had been implemented.The method also provides test data related to the fictitious trafficsituation. The vehicle function is operated passively here in thevehicle to determine the test control instruction.

A disadvantage of this method is that an actual operation of the vehicleis required for the verification and/or validation of the vehiclefunction in order to determine the required data.

An autonomous vehicle contains a large number of control units. Eachindividual control unit and its interconnection must be extensivelytested during development, function release, and homologation.Scenario-based testing can be used to ensure the faultless functioningof the control units in any traffic situation. In scenario-basedtesting, the vehicle's driving style is analyzed in a traffic situationthat is as close to reality as possible. The traffic situation aspectsto be analyzed and their evaluation depend on the system to be tested.To this end, scenarios that can be described as an abstraction of atraffic situation are defined in the scenario-based testing of systemsand system components for the autonomous guidance of a motor vehicle. Alogical scenario here is the abstraction of a traffic situation with theroad, the driving behavior, and the surrounding traffic withoutspecifying concrete parameter values. By choosing concrete parametervalues, the logical scenario becomes a concrete scenario. Such aconcrete scenario corresponds to a single traffic situation in eachcase. Test cases can then again be executed for each scenario. Acombination of test cases corresponds to a test suite.

A deviating specification is required to vary the test cases, testsuites, and/or scenarios.

For this purpose, parameter combinations can be selected independentlyby algorithms or manually by a test engineer. Therefore, in order tooptimally support the specification, a computer-implemented method forthe use of stored specification parts is provided according to theinvention, in order to be able to achieve a test process that is asresource-saving and time-reduced as possible.

SUMMARY OF THE INVENTION

It is therefore an object of the invention to provide a method and adevice that advances the prior art. To this end, structures must becreated to store, manage, and logically retrieve specification parts sothat they can be incorporated into further specifications.

In accordance with an exemplary embodiment of the invention, acomputer-implemented method is proposed for the use of storedspecification parts of at least one test and/or simulation, comprisingthe following steps: providing the at least one test to be specifiedand/or the one simulation for testing driving functions of a vehicle andthe at least one test and/or the at least one simulation are determinedby at least one parameter value and/or setting value; and performing aspecification of the at least one test and/or the at least onesimulation, wherein the specification includes specification parts,wherein the parameter values and/or the setting values are selected andconcrete parameter values and/or setting values are assigned by thespecification for the test and/or the simulation, wherein the parametervalues and/or setting values are selected manually or automatically, andwherein already stored specification parts are selected manually and/orautomatically and integrated into the specification.

For the basic inventive differentiation of traffic scenarios orscenarios in scenario-based testing, not only static parameters, suchas, e.g., but not limited to surroundings, buildings, or roadway width,are used, but also in particular the driving behavior of the individualroad users. The movements of the road users and thus the drivingbehavior are described by trajectories. Trajectories describe a path inboth spatial and temporal directions. Movements of road users can bedifferentiated by parameters such as, e.g., speed.

An autonomous driving function can be realized by a system, for example,a control unit. The control unit can be conventionally tested in thereal vehicle in real traffic situations and validated byhardware-in-the-loop tests or alternatively by fully virtual tests.

A simulation can also be used for this purpose. In addition toscenario-based testing, other virtual testing methods can also be used,for example, step-based testing, replay testing, or requirement-basedtesting. Scenario-based testing is carried out here merely as anexample.

Using the present method, for example, a so-called cut-in scenario canbe differentiated from other scenarios. The cut-in scenario can bedescribed as a traffic situation in which a highly automated orautonomous vehicle is driving in a predetermined lane and anothervehicle with a reduced speed compared to an ego vehicle cuts fromanother lane into the lane of the ego vehicle at a certain distance. Theego vehicle here refers to the vehicle under test (SUT).

In general the term “ego vehicle” can represent a virtual vehicle in thecenter of a simulation or a test. E.g. the vehicle for that a newfunction is to be developed or tested. Typically, one skilled in the artuses such to distinguish a central vehicle (“ego”) from other vehiclesor traffic participants (pedestrians, bicycles, etc.) that are usuallycalled “fellows” or “fellow vehicles” that appear in a simulation ortest and can interact or have an impact on the ego. For example, theremay be several vehicles in a scenario in order to test a function of theego vehicle but these fellow vehicles may not have the function to betested, e.g. automatic braking systems.

The speeds of the ego vehicle and the other vehicle, which is alsocalled the fellow vehicle, are constant in this case. Because the speedof the ego vehicle is higher than that of the fellow vehicle, the egovehicle must be slowed down to avoid a collision of the two vehicles.

But a cut-in scenario can also take various forms, such as, e.g., adifference in the speed of the road users.

Furthermore, a scenario such as a passing-by scenario is possible inthat the ego vehicle represents the overtaking vehicle, so that the egovehicle travels in a predetermined lane and overtakes another vehiclewith a reduced speed compared to the ego vehicle. In this case, the egovehicle changes/swerves to another lane and passes the fellow vehicle ata higher speed. The speed of the ego vehicle does not have to beconstant in this scenario. After the ego vehicle has passed the fellowvehicle, the ego vehicle swerves back into the previous lane.

Such scenarios can be simulated with different parameter values. Inorder to use simulation and computing time in the most resource-savingway, a test follow-up control is required to achieve highcoverage—homologation. Likewise, a reuse of specifications and/orspecification parts that have already been developed is necessary forthis purpose.

For this purpose, deviating types of parameters that usually have to bespecified in advance are listed: scenery parameters comprising at leastone of the features: a number of and/or a width of a lane and/or curvesand/or road restrictions and/or an ambient temperature; and drivingsituation parameters describing the number and properties of movingobjects in the scenario, comprising at least one of the features: anumber of road users and/or a number of lane changes in a trafficsituation and/or a speed of the road users and/or transport vehicle.

In addition, setting values can be specified, which are used inparticular for monitoring the selected parameters. Such setting valuesor also called observers can check whether parameters exceed a thresholdvalue, whether parameters lie within a defined value range for a periodof time, or also whether combinations of parameters lead to a definedstate. Such setting values must also be specified. Reusing thesespecification parts for further future tests can significantly increaseefficiency during testing.

In this regard, a specification of a test, a scenario of a simulation,or a combination of tests can be stored and versioned as a whole or onlyparts of the specification, therefore, specification parts. Allspecifications and/or specification parts are provided withmeta-information. The contents of the specification are indicated by themetadata/meta-information, so that a user and/or a system can findspecification parts more easily. According to the invention, thismeta-information constitutes elements of an operational design domain.An operational design domain describes an application context of aproduct and thus defines uniform terms to describe it. In this regard, aspecification part can be assigned one and/or more meta-information.

New specification parts created during the specification of a testand/or simulation are saved, versioned, and stored in a library. Thiscan be located locally on a computer, in a server environment such as,for example, a data center, or in a cloud environment.

The specification itself can be carried out by a user in a graphicaluser interface. According to the invention, a text editor can be used asa further input. Different levels of user knowledge can also be takeninto account by this and the efficiency can be increased overall. Theincorporation of stored specification parts in both variants, thegraphical user interface or text editor, is possible. A testing and/orsimulation tool can also suggest and integrate/incorporate storedspecification parts. The retrieval is realized by the specifications ofthe meta-information. This further accelerates the testing and/orsimulation process.

The storage and versioning of the specification parts of a test and/orsimulation also enables the exchange to third parties, such as, e.g.,other development teams and/or clients. In order to enable a protectedexchange, an encryption of the specification parts is to be realizedaccording to the invention. Thus, there will then be a key forencryption and another key for decryption, which is transmittedseparately. Both an encrypted and non-encrypted exchange ofspecification parts is provided, depending on the parties involved andthe degree of confidentiality required.

According to an additional aspect of the invention, further a computerprogram is provided with a program code to carry out the method of theinvention when the computer program is executed on a computer. Accordingto an additional aspect of the invention, a data storage medium isprovided with a program code of a computer program in order to carry outthe method of the invention when the computer program is executed on acomputer.

Further scope of applicability of the present invention will becomeapparent from the detailed description given hereinafter. However, itshould be understood that the detailed description and specificexamples, while indicating preferred embodiments of the invention, aregiven by way of illustration only, since various changes, combinations,and modifications within the spirit and scope of the invention willbecome apparent to those skilled in the art from this detaileddescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description given hereinbelow and the accompanying drawingswhich are given by way of illustration only, and thus, are not limitiveof the present invention, and wherein:

For a better understanding of the present invention and its advantages,reference is now made to the following description in conjunction withthe accompanying drawings. The invention will be explained in moredetail hereinbelow with reference to exemplary embodiments which areshown in the schematic illustrations of the drawings.

FIG. 1 shows a schematic diagram for differentiating scenarios accordingto the invention;

FIG. 2 shows another schematic diagram for differentiating scenariosaccording to the invention;

FIG. 3 shows a schematic view indicating a boundary between critical andnoncritical test results;

FIG. 4 shows a sequence of the invention for the use of storedspecification parts;

FIG. 5 shows two schematic views of a library for managing storedspecification parts;

FIG. 6 shows a schematic view of the use of metadata in the method ofthe invention;

FIG. 7 shows schematic views of user interfaces on the device of theinvention, as well as the application of the method; and

FIG. 8 shows a sequence for the use of encryption in the method of theinvention.

DETAILED DESCRIPTION

FIG. 1 shows a schematic diagram for scenario differentiation of theinvention. On the left side of FIG. 1 a scenario is depicted which showsa turning maneuver and has one ego vehicle and four fellow vehicles. Onthe right side of FIG. 1 , the same intersection area is shown butwithout a turning maneuver with only one ego vehicle.

Different scenarios are shown by FIG. 1 which allow different parametercombinations and also enable different critical test cases. In ascenario-based testing process of a driving assistance system, theidentification and specification of critical test cases in particular,for example, a vehicle collision or a near collision, are of importance.

This means that in order to derive a concrete test case, a specificationof the scenarios is necessary. Specification parts can be reused here,especially to increase the efficiency of the testing process.

FIG. 2 shows a further schematic diagram for the differentiation ofscenarios according to the invention (S1 and S2). According to FIG. 2 ,the scenarios can be completely different with respect to parametervalues and/or setting values, have overlapping parameter values and/orsetting values, or be the same with respect to the subset of theirrespective parameter values and/or setting values. Specification partscan be reused here as well. These specification parts must bespecifically stored and versioned for this purpose.

FIG. 3 shows by way of an example the cut-in scenario using the drivingsituation parameters VEGO, i.e., a speed of the ego vehicle, and on thevertical axis VFELLOW, i.e., the speed of the fellow vehicle in front.

The function shown in FIG. 3 forms the boundary between critical andnoncritical test results. The points shown are approximated testresults. Alternatively, the points shown can be simulated test results,for example. For this purpose, corresponding parameter values and/orsetting values had to be specified so that these test results areobtained.

The evaluation function shown is the safety objective function, whichhas a numerical value that has a minimum value when a safety distancebetween the motor vehicle and the further motor vehicle is≥VFELLOW×0.55,has a maximum value when there is a collision between the motor vehicleand the further motor vehicle, and has a numerical value that is greaterthan the minimum value when the safety distance between the motorvehicle and the other motor vehicle is≤VFELLOW×0.55. In particular, suchmonitoring setting variables must be specified in advance, but can beexpediently reused for further tests. For this purpose, appropriatespecification parts must be stored and versioned so that reuse ispossible.

As an alternative to the safety objective function, a comfort objectivefunction or an efficiency objective function can be approximated, forexample, which has a numerical value that has a minimum value in thecase of no change in the acceleration of the motor vehicle, has amaximum value when there is a collision between the motor vehicle andthe further motor vehicle, and has a numerical value between the minimumvalue and the maximum value when there is a change in the accelerationof the motor vehicle as a function of the amount of the change in theacceleration.

A large number of parameter values often have to be defined for thespecification of such test scenarios. In addition, setting values mustbe found that allow monitoring and thus subsequent analysis of thetesting process. Such setting values must also be determined and definedas part of the specification. In the example given, in addition to thescenario-determining parameter values, such as speeds of the ego andfellow vehicles, e.g., the time-to-collision, therefore, the timeremaining until the collision as a monitoring variable, can be definedas a setting value.

FIG. 4 shows a sequence of the invention for the use of storedspecification parts. For this purpose, as a first step (D1), at leastone scenario to be specified, test, and/or simulation are provided atfirst. These scenarios, tests, and/or simulations are determined by atleast one parameter value or setting value. In this case, parametervalues describe the framework conditions to be tested, such as, forexample, the speeds of the vehicles involved, lane widths, or roadcourses. Monitoring variables are defined with the setting values sothat the test or simulation can be sufficiently monitored and analyzed.

The specification of the at least one scenario, test, and/or simulationis carried out as a second step (D2). For this purpose, parameter valuesor setting values are described concretely and assigned actualquantities. For this purpose, stored specification parts can be accessedfrom a library (L). These specification parts can be integrated into theoverall specification. This saves considerable resources that wouldotherwise be used to repeatedly determine parameter values or settingvalues. Some variables are to be used in multiple tests or to monitorcorresponding variables. This also applies to different tests,scenarios, or simulations. Therefore, reuse is desirable. For thispurpose, the specific storage including versioning and the use of storedspecification parts is useful. The selection of stored specificationparts can be done manually by a user or automatically by a testingand/or simulation tool. If new specification parts not yet stored ariseduring the specification (D2), these specification parts can be added tothe library (L) and stored.

Finished specified tests, scenarios, or simulations can then be executedvia the testing and/or simulation tool (E).

FIG. 5 shows two schematic views of a library (L1 and L2) for managingstored specification parts. Corresponding libraries (L) can bestructured differently. The library (L1) shown on the left of the figuredisplays specification parts (SP) managed in a tree structure. Moreover,this makes it easy to navigate to corresponding specification parts (SP)and a quick overview of existing specifications can be ensured.

On the right side of the figure, a library (L2) is shown, which is to beunderstood more as a pool. Specification parts (SP) are stored collectedtogether here. For better findability, specification parts (SP) areassigned meta-information (T) or also tags. Moreover, specificationparts (SP) can be searched for in a targeted manner. Such a search canbe performed by a user/operator but also by a testing and/or simulationtool, so that automatic offering of specification parts (SP) that may berequired is also possible.

Further embodiments or combinations of embodiments with respect to thedesign of the library (L) are conceivable and are included according tothe invention.

FIG. 6 shows a schematic view of the use of metadata/meta-information(T) in the method of the invention.

Meta-information can be selected as free text or offer already definedterms. For this purpose, the use of elements of an operational designdomain is proposed according to the invention.

An operational design domain (ODD) for an at least partially autonomousvehicle is defined by SAE J3016 (2021) as: “Operating conditions underwhich a given driving automation system, or feature thereof, isspecifically designed to function, including, but not limited to,environmental, geographical, and time-of-day restrictions, and/or therequisite presence or absence of certain traffic or roadwaycharacteristics.”

Thus, the ODD describes the limits within which the at least partiallyautonomous vehicle is to be operated and, as such, operates only whenthe parameters described in the ODD are met. Such an ODD is designed forat least partially autonomous vehicles and can serve as a basis fordefining meta-information (T) for the method of the invention.

One of the scenarios from FIG. 1 is shown on the left side in FIG. 6 .For this purpose, a possible excerpt from the ODD is shown. Here, forexample, the number of lanes on each side (#L), information about leftor right traffic (L/R), as well as the number of fellow vehicles (F) canbe included in the scenario. Thus, a specification part (SP) can beassigned no meta-information (T) but also likewise a multitude ofmeta-information (T). The better information about a specification isstored here, the easier it is to find it manually and/or automaticallyand to integrate it into a new specification.

Further embodiments are possible and are covered according to theinvention.

FIG. 7 shows schematic views of user interfaces on the device of theinvention, as well as the application of the method. In this regard,various embodiments of a user interface are covered according to theinvention.

A graphical user interface is possible as shown in FIG. 7 on the left,where a user can specify in a guided manner parameter values for a test,scenario, and/or simulation. Here, a user can also manually search forstored specification parts (SP) or automatically receives specificationparts (SP) suggested by the testing and/or simulation tool.

A text-based editor can be seen in the middle view in FIG. 7 . This canalso be used to specify tests, scenarios, or simulations. Specificationsor specification parts that have already been saved can again besearched for manually or displayed automatically and integrated into thespecification using common commands. It should also be possible toselect and save specifically specification parts and to version them.

As shown in the right view in FIG. 7 , a new text-based specification isalso possible without recourse to already existing specification parts.

In addition, further embodiments are possible and are intended to becovered by the method and device of the invention.

FIG. 8 shows a sequence for using encryption in the method of theinvention.

If, in the course of the specification, a new specification part thathas not yet been saved should arise, this can be saved and versioned, onthe one hand, and also encrypted, on the other. Encryption allows thespecification parts to be passed on to third parties without releasingthe source code or realizes a secure transfer to the third party.

First, it is decided that the corresponding specification part is to besaved. Then meta-information can be added to the correspondingspecification part (V1), as already described. Subsequently, it isdetermined whether the specification part is fed directly into thelibrary (L) or whether an encryption (V2) is initiated first. Knownencryption methods can be used for this purpose. An encryptedspecification part can only be integrated into another specification ifthe required key to release the specification part is known. Rights tothe specification can be preserved in this way or new business modelscan also be established.

The invention being thus described, it will be obvious that the same maybe varied in many ways. Such variations are not to be regarded as adeparture from the spirit and scope of the invention, and all suchmodifications as would be obvious to one skilled in the art are to beincluded within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method for storedspecification parts of at least one test and/or simulation, the methodcomprising: providing the at least one test to be specified and/or theone simulation for testing driving functions of a vehicle and the atleast one test and/or the at least one simulation are determined by atleast one parameter value and/or setting value; and performing aspecification of the at least one test and/or the at least onesimulation, the specification including at least one specification part,selecting parameter values and/or setting values; and assigning concreteparameter values and/or setting values by the specification for the testand/or the simulation, the parameter values and/or setting values areselected manually or automatically, wherein already stored specificationparts are selected manually and/or automatically and integrated into thespecification.
 2. The computer-implemented method according to claim 1,wherein the method further comprises at least one test of a device forthe at least partially autonomous guidance of a transport vehicle,wherein the transport vehicle comprises at least one ego vehicle and/orfellow vehicles, and wherein the ego vehicle is a vehicle with a systemunder test and a fellow vehicle is any other vehicle.
 3. Thecomputer-implemented method according to claim 1, wherein the parametervalues comprise: scenery parameters comprising at least one of thefeatures: a number of and/or a width of a lane and/or curves and/or roadrestrictions and/or an ambient temperature; and/or driving situationparameters describing the number and properties of moving objects in thescenario, comprising at least one of the features: a number of roadusers and/or a number of lane changes in a traffic situation and/or aspeed of the road users and/or transport vehicle.
 4. Thecomputer-implemented method according to claim 1, wherein the settingvalues comprises setting values for monitoring at least one selectedparameter and/or parameter value during the test execution/simulation.5. The computer-implemented method according to claim 1, wherein thespecification of the at least one test and/or the at least onesimulation has at least one specification part and each specificationpart is stored separately and provided with meta-information.
 6. Thecomputer-implemented method according to claim 5, wherein themeta-information indicates the relationship between a specification partand an element of an operational design domain, and wherein an elementof an operational design domain provides one or more elements of thescenario under test, and the operational design domain thus comprisespredefined elements that are used as meta-information for specificationparts.
 7. The computer-implemented method according to claim 5, whereinthe stored specification parts are found by their meta-informationmanually by a user of a testing and/or simulation tool and/orautomatically by a testing and/or simulation tool itself.
 8. Thecomputer-implemented method according to claim 1, wherein a newlycreated specification part is stored and the stored specification partis stored and versioned locally and/or in a server environment and/or ina cloud environment, and wherein the stored specification part ismanaged in a library.
 9. The computer-implemented method according toclaim 1, wherein the execution of the specification by storedspecification parts can be done by an input in a text editor and/or byinput in a graphical user interface of the testing and/or simulationtool.
 10. The computer-implemented method according to claim 1, whereina release of the integration of existing specification parts takes placeby entering and/or transferring a key, wherein one party uses a key forencryption and another and/or the same party uses another key fordecryption, and wherein the key for decryption is transmittedseparately.
 11. The computer-implemented method according to claim 1,wherein the encryption of the specification parts enables an exchange ofthese specification parts and prevents disclosure of the contents of thespecification parts between the parties and/or to third parties.
 12. Atesting and/or simulation tool set up for stored specification parts ofat least one test and/or simulation, the tool comprising the followingsteps: providing the at least one test to be specified and/or the onesimulation for testing driving functions of a vehicle and the at leastone test and/or the at least one simulation are determined by at leastone parameter value and/or setting value; and performing a specificationof the at least one test and/or the at least one simulation, wherein thespecification comprises at least one specification part, wherein theparameter values and/or the setting values are selected and concreteparameter values and/or setting values are assigned by the specificationfor the test and/or the simulation, wherein the parameter values and/orsetting values are selected manually or automatically, and whereinalready stored specification parts are selected manually and/orautomatically and integrated into the specification.
 13. The testingand/or simulation tool according to claim 12, wherein the testing unitis formed by a control unit for which at least one simulation and/or atleast one virtual and/or real test is defined.
 14. A computer programwith a program code to perform the method according to claim 1, when thecomputer program is executed on a computer.
 15. A non-transitorycomputer-readable data storage medium containing the program code of acomputer program in order to carry out the method according to claim 1,when the computer program is executed on a computer.